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SJMARY 


Scaling  factors  in  raultiattribute  utility  measurement  can 
either  be  assessed  directly  as  importance  weights  or  indirectly 
by  indifference  judgments.  Critics  of  the  importance  weight  inter- 
pretation of  scaling  factors  argue  that  importance  weights  are  not 
sensitive  to  ranges  of  alternatives  and  thus  cannot  be  used  to 
match  standardized  single  attribute  utility  functions.  To  exa- 
mine the  range  sensitivity  of  importance  weight  judgments  two 
experiments  were  designed.  In  the  first  experiment  college  students 
gave  relative  importance  weight  judgments  for  a number  of  attri- 
butes when  evaluating  apartments  and  liquiiied  natural  gas  plant 
locations.  After  the  initial  importance  weight  assessments  the 
range  of  alternatives  in  one  attribute  was  changed  and  subjects  reas- 
sessed their  weights.  Although  subjects  were  explicitly  in- 
structed to  take  ranges  into  account  when  making  these  judgments, 


they  were  unable  to  adjust  their  weights  appropriately.  To  mag- 
nify possible  range  effects  a second  experiment  examined  a very 
simple  two  attribute  car  evaluation  problem.  Subjects  were  asked 
directly  if  weights  should  change  after  the  range  in  one  attribute 
was  doubled.  Most  subjects  indicated  that  there  should  be  no  change. 
The  results  of  these  experiments  suggest  that  subjects  have  plausi- 
ble ranges  in  mind  when  assessing  importance  weights  and  that  they 


are  unwilling  to  change  weights  after  relatively  spurious  changes 
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Introduction 


Multiattribute  Utility  Theory  (MMJT)  forms  a class  of  models 
and  scaling  procedures  for  evaluating  complex  multidimensional 
alternatives,  such  as  apartments,  industrial  sites,  or  social 
programs.  The  procedure  involves  three  steps.  First  is  the 
identification  of  the  possible  alternative  set.  The  second  step 
involves  the  construction  of  utility  functions  to  evaluate  each 
alternative  with  respect  to  each  attribute.  Finally  these  single 
attribute  utility  functions  are  combined  by  an  appropriate  ag- 
gregation rule.  The  best  known  aggregation  rule  is  the  additive 
model  (see  Keeney  and  Raiffa,  1976;  and  Edwards,  1977).  According 
to  the  additive  model,  single  attribute  utility  functions  (u^)  are 
multiplied  by  a weight  (w^  and  additively  combined: 

n 

u(x)  •=  I W.u.(x.)  (1) 

i*l 

where  x is  an  alternative  under  evaluation  and  x^  is  the  level  of 
that  alternative  on  attribute  i.  Decision  analysts  usually  stand- 
ardise single  attribute  utility  functions  to  range  between  0 
(worst  level)  and  100  (best  level);  and  normalize  weights  to  sum 
to  one. 

In  riskless  applications  of  this  model,  the  utility  functions 
reflect  the  value  (or  worth)  differences  within  an  attribute.  If, 
for  example,  u^(xp  ■ 50,  then  the  worth  of  x^  is  halfway  between 
the  best  and  the  worst  level  in  attribute  i.  The  weighting  factors 
w^  match  utilities  across  attributes.  Such  a match  is  necessary 
since  not  all  attributes  contribute  equally  to  overall  worth. 
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Consider,  for  exanple,  the  attributes  "rent"  and  "distance  from 
work"  in  an  apartment  evaluation  problem.  Assume  that  for  you  a 
rent  decrease  from  $300  to  $280  is  just  as  valuable  as  a distance 
decrease  from  15  to  5 miles.  Further  assume  that  the  single  dimen- 
sion utility  flections  are  standardized  so  that  the  $20  rent  de- 
crease corresponds  to  a gain  of  10  utility  points  (out  of  100  a- 
vailable)  while  the  10  mile  distance  decrease  corresponds  to  a gain 
of  50  points.  To  make  up  for  this  distortion  of  single  attribute 
utilities,  the  attribute  "rent"  has  to  be  weighted  five  times 
more  than  the  attribute  "distance."  Weights  thus  spell  out  how  much 
a utility  unit  in  one  attribute  contributes  to  overall  worth  relative 
to  a unit  in  another  attribute. 

This  interpretation  as  relative  scaling  factors  makes  weights 
directly  dependent  on  the  relatively  arbitrary  choice  of  a unit 
of  single  attribute  utility  functions,  and,  in  particular,  on 
the  choice  of  attribute  ranges.  If,  for  exauple,  in  the  above 
apartment  evaluation  problem  15  and  5 miles  had  been  the  worst  and 
the  best  level  of  the  attribute  "distance",  the  utility  difference 
would  have  been  100  rather  than  50  and  the  weight  of  "rent"  should 
have  been  10  times  larger  than  the  weight  of  "distance." 

Procedures  to  construct  additive  multiattribute  utility  func- 
tions should  therefore  reflect  this  sensitivity  to  choices  of  units 
and  ranges.  Classical  indifference  methods  such  as  standard  sequen- 
ces (see  Krantz,  Luce,  Suppes  and  Tversky,  1971)  handle  this  sensi- 
tivity by  matching  units  of  single  attribute  utility  functions  di- 
rectly in  the  constructive  process.  The  results  of  standard  sequence 


procedures  are  properly  fetched  utility  functions  which  need  no 
further  weighting  (and,  of  course,  do  not  range  between  0 and  100). 
However,  such  indifference  methods  involve  hypothetical  trade-off 
questions  which  are  sometimes  conplex,  often  difficult  to  under- 
stand, and  seldom  realistic. 

The  sinpler  and  more  intuitively  understandable  magnitude 
estimation  techniques,  on  the  other  hand,  may  suffer  from  an  in- 
sensitivity to  the  choice  of  units  and  ranges.  Edwards'  SMART 
procedure  (1977)  i_  the  best  kncwi  exarple  of  nuch  magnitude  esti- 
mation techniques.  In  this  procedure  single  attribute  utilities 
are  scaled  from  0 to  100,  where  the  endpoints  represent  the  best 
and  worst  available  alternatives.  Judging  the  relative  importance 
of  one  attribute  over  another  provides  mmerical  assessment  of 
weights.  The  range  sensitivity  of  the  numerical  judgments  of  "re- 
lative importance"  is,  however,  not  at  all  obvious.  In  fact,  the 
meaning  of  "importance"  suggests  some  degree  of  situational  invari- 
ance, as  in  statements  such  as  "money  is  always  the  most  important 
consideration."  Sophisticated  applications  of  SMART  try  to  take 
this  problem  into  account  by  making  ranges  explicit  in  the  import- 
ance weight  assessment.  For  example,  in  so  called  "swing  weight" 
assessment  judges  state  how  much  more  important 
a step  from  the  ’ orst  to  the  best  alternative  in  attribute  i is 
relative  to  a step  from  the  worst  to  the  best  alternative  in  attri- 
bute j.  Obviously  such  refinements  intend  to  make  the  ijiportance 
weight  assessment  range  sensitive. 

Knowing  that  some  ambiguity  surrounds  the  notion  of  importance 
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weighting,  the  present  study  asked  whether  subjects  are  range  sensi- 
tive in  the  SMART  weighting  techniques.  The  normative  rules  for  chan- 
ging the  weight  as  a function  of  the  range  change  are  explored. 

The  description  of  two  experiments  which  tested  the  range  sensitivity 
of  the  SMART  weighting  technique  is  also  included.  In  the  first 
experiment,  subjects  applied  the  full  SMART  procedure  in  an  apart- 
ment selection  problem,  and  in  the  siting  of  a liquified  natural  gas 
plant.  The  subjects  then  repeated  the  evaluations  with  a different 
range  on  one  attribute  of  each  problem.  The  second  experiment  at- 
tested to  magnify  the  range  sensitivity  in  a simple  two  attribute 
car  evaluation  problem  which  required  subjects  to  explicitly  change 
weights . 


How  Should  Weights  Be  Changed  If  the  Range  Changes? 

The  introductory  apartment  evaluation  exanple  clearly  shows  that 
weights  are  dependent  on  the  choice  of  units  and  ranges  of  single  at- 
tribute utility  functions.  Res tandardi zing  single  dimension  utility 
functions  or  changing  units  requires  changing  weights.  As  a rule,  the 
weight  ratios  change  inversely  proportional  to  the  change  of  the  unit. 
If  a range  decrease,  for  exarple,  enlarges  the  unit  of  a utility  func- 
tion by  201  the  weight  ratios  between  this  utility  function  and  others 
should  be  201  smaller.  In  other  words,  a change  in  the  total  utility 
range  should  result  in  a proportional  change  of  the  (non- renormalized) 
revised  weight.  Of  course,  arter  renormalization,  all  weights  will 
change . 

More  formally,  assure  that  model  (1)  is  an  appropriate 
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evaluation  model,  with  properly  constructed  and  scaled  weights  w^ 
and  single  attribute  utility  fuictions  u^.  Further,  assune  that 
in  one  attribute,  say  Xj,  a change  in  range  occurs.  For  illus- 
tration, Figure  1 shows  a range  decrease  of  an  attribute  both 
at  its  upper  and  lower  end.  Restandardizing  leads  to  a revised 
utility  function  uj'.  How  should  the  weights  w^  change  to  weights 

w .'? 

1 

Since  both  Uj  and  u^'  preserve  utility  difference  judgments, 
the  following  equation  must  hold  for  any  two  attribute  levels  Xj 
and  yj,  for  which  both  and  Uj'  are  defined: 

W'VyP  - TtUj'fXjj-Uj’tyj)]  (2) 

Multiplying  both  sides  with  the  original  weight  Wj  gives 

W1[Ul*X1-*U1(YpJ  - (TWj) [uj ' (Xj) -Uj ' (Vj)]  (3) 

Before  the  range  change  ^ natched  a utility  irit  in  u.  against  a 

w. 

i 

utility  unit  in  any  other  attribute  i.  As  (3)  shows  T matches  a 

utility  nit  in  Uj  against  a utility  init  of  the  changed  utility 

Tw 

function  u '.  Consequently  1^  matches  a utility  unit  in  u^'  a- 

wi 

gainst  a utility  unit  in  any  other  attribute  i.  Algebraic  mani- 
pulation of  (2)  gives: 

T - (4) 

Uj'  (Xjj-Uj'  (y^ 

for  any  two  points  which  u^  and  u^  share.  Consequently  the  non- 
normalized  revised  weight  w,"  should  be 


UjUj)  - Uj(yj) 


(5) 
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Observing  that  by  the  original  normalization  of  the  w/s 


1-w, 


n 

I w. , 
i-2  1 


the  nonnalized  revised  weights  w.  ' follow  as 

*1 


V "t vnr^r 


(6) 

(7) 


and 


w. 

l 


w. ' 


i " “TTwjl 


, i > 1. 


(8) 


In  general,  a change  of  the  raige  or  units  in  all  n 

utility  fuictions  changes  the  weights  to 

T.w. 

....  ii 


n 

r Tjwj 


(9) 


3*1 


This  normative  revision  of  weights  gives  rise  to  three  observa- 
tions. First,  T should  be  smaller  than  1 if  the  range  decreases,  and 
larger  than  1 if  it  increases.  In  fact,  T gives  the  factor  by  which 
a utility  range  decreases  or  increases  relative  to  the  original  uti- 
lity range.  Second,  the  change  in  the  non-normalized  weight  w^" 
and  consequently  in  the  revised  weight  ratios  w^'/V  is  proportional 
to  T.  Finally,  the  nonnalized  revised  weights  w^'  do  not  change  in 
proportion  to  T. 

Otway  and  Edwards  (1977)  used  equation  (9)  to  recalculate  weights 
in  a nuclear  waste  disposal  evaluation  problem.  In  that  application 
experts  assessed  weights  based  on  ranges  which  were  often  substantially 
larger  than  those  of  the  actual  sites  to  be  evaluated.  Since  single 
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attribute  utility  finctians  u.  were  linear,  equation  (9)  was  the 
obvious  candidate  to  confute  normatively  correct  transformations 
of  the  weights  assessed  by  the  experts.  As  an  alternative  the  ex- 
perts could  have  reassessed  their  own  weights  based  on  the  smaller 
ranges.  Would  that  reassessment  agree  with  the  normative  rule? 

This  question  is  addressed  in  the  following  experiment. 

EXPERIMENT  1 
Method 

Subjects.  TVenty-four  undergraduate  psychology  students  at  the 
University  of  Southern  California  participated  as  subjects.  All 
of  them  received  experimental  credit  which  they  used  in  partial 
fulfillment  of  their  introductory  psychology  course  requirements. 
Participants  signed  ip  on  a first  come  basis. 

Stimuli.  A car  evaluation  example  introduced  subjects  to  multi- 
attribute evaluation  problems  and  to  the  SMART  procedure.  Stimuli 
were  5 cars  described  on  six  dimensions  as  in  Table  1.  Test  sti- 
muli were  5 apartments  described  on  six  attributes  and  5 sites  for 
Liquified  Natural  Cas  (LNG)  plants  also  described  on  six  attri- 
butes (see  Tables  2 and  3).  In  the  test  tasks  one  attribute  range 
was  variable.  The  changed  attribute  values  are  given  in  parenthesis 
in  Tables  2-3.  In  the  automobile  and  the  apartment  evaluation 
problem  subjects  had  to  make  judgments  according  to  their  own  pre- 
ferences. In  the  LNG  plant  siting  problem  subjects  had  to  consider 
the  preferences  of  a governmental  decision  maker  responsible  for 
selecting  a proper  site. 

Procedure.  Subjects  received  written  instructions  to  use  the  SMART 
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TABLE  1 

Stimuli  for  the  Car  Evaluation  Exanple 

CARS 


A 

B 

C 

D 

E 

Gas  mileage 

(average  miles  per  gallon) 

10 

20 

30 

40 

SO 

Price 

(in  thousands  of  dollars) 

5 

4.5 

6.2 

7.0 

3.5 

Rear  knee  room 
(in  inches) 

IS 

14 

12 

20 

19 

Iapact  speed  that  the  front 
end  will  withstand  leaving 
the  passenger  compartment 
intact  (in  miles  per  hour) 

35 

40 

37 

39 

29 

Tnnk  size 
(in  cubic  feet) 

9 

12 

20 

16 

19 

Interior  noise 

70 

74 

76 

75 

74 

(in  dhA  at  50  nph) 


WWHCWM»HH> 


TABLE  2 

Stimuli  for  the  Apartment  Evaluation 
Problem  (In  Brackets:  Short  Range) 
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SITES 


A 

B 

C 

D 

E 

* 

Break-ins  per  year 

20 

15 

10 

5 

0 

(15) 

(10) 

(5) 

(1) 

(0) 

Rental  fee  in  constant 
dollars 

200 

325 

150 

210 

305 

Miles  to  work 

15 

25 

23 

7 

10 

Age  of  complex  in  years 

10 

5 

0 

20 

28 

Number  of  bedrooms 

2 

1 

2 

1 

•* 

0 

Number  of  canplexes  in  area 

2 

9 

1 

2 

20 

* 

in  neighborhood 

** 


studio  apartment 
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TABLE  S 

Stimuli  for  the  LNG  Plant  Siting 
Problem  (In  Brackets:  Short  Range) 


SITES 


A 

A 

B 

C 

D 

E 

T 

Environmental  damage  in 
millions  of  dollars 

0 

4 

8 

12 

16 

T 

(0) 

(4) 

(6) 

(8) 

(12) 

R 

I 

B 

U 

T 

Economic  inpact  in  millions 
of  dollars 

80 

90 

100 

70 

60 

Miles  to  port 

5 

7 

3 

0 

9 

Years  of  operation 

20 

10 

30 

40 

50 

Nunber  of  faults 

6 

7 

5 

9 

8 

1 

E 

Population  density  in 
people  per  square  mile 

100 

75 

125 

50 

25 

S 


I 
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procedure.  The  instructions  included  a detailed  exanpie  of  an  auto- 
mobile evaluation  problem.  The  subjects  examined  Table  1 and  then  saw 
how  attribute  levels  could  be  rescaled  based  on  the  value  levels  for 
each  automobile  on  each  attribute.  The  five  automobiles  were  rank  or- 
dered according  to  their  levels  on  the  attribute  in  question.  After 
setting  the  best  value  of  each  attribute  equal  to  100  and  the  worst 
equal  to  0,  subjects  then  learned  hew  intermediate  levels  can  be  set 
to  correspond  to  their  relative  value  judgments. 

The  use  of  the  SMART  procedure  provided  training  for  the  subjects 
in  the  assignment  of  iirportance  weights.  Subjects  saw  how  the  six 
attributes  can  be  rank  ordered  by  importance  with  the  arbitrary  as- 
signment of  10  to  the  least  important  dimension.  Weights  for  the  oth- 
er dimensions  are  then  assigned  as  multiples  of  the  least  important 
dimension.  Subjects  were  clearly  instructed  to  take  ranges  into  ac- 
count when  making  relative  importance  judgments: 

"for example,  you  might  consider  "trunk  sizeTto  be  five  times  as 
inportant  as  (the  least  important  dimension)  "interior  noise"  (va- 
lue * 10).  In  this  case  you  would  assign  the  value  50  (5  times 
10)  to  the  factor  "trunk  size."  By  assigning  a weight  of  50 

to  one  attribute  (as  in  the  case  of  "trunk  size")  you  are  saying 
that  a certain  reduction  (say  101)  in  the  location  measures  fa- 
tuity) for  the  attribute  is  equivalent  to  5 times  10  or  501  re- 
duction in  the  location  measures  for  the  attribute  with  the  weight 
of  10.  In  other  words,  the  relative  magnitude  of  the  numbers  re- 
flect how  a 101  reduction  of  the  worth  (the  location  measure)  of 
the  values  in  one  attribute  conpares  with  the  same  reduction  on 
the  other  attributes.  For  instance,  the  weight  50  on  the 
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attribute  "trunk  site"  Beans  that  losing  10%  on  tnnk  size  is 
five  times  as  bad  as  losing  10%  on  interior  noise.  You  may  re- 
examine your  ranking  in  light  of  this  consideration." 

Based  on  this  exanple  of  assessing  utility  functions  and  impor- 
tance weights,  subjects  applied  SMART  to  the  apartment  and  the  LNG 
siting  problems.  After  completing  both  problems  subjects  saw  es- 
sentially the  same  problems  again,  but  this  time  the  range  in  the 
first  attribute  ("nunber  of  break-ins"  in  the  apartment  example; 
"environmental  damage"  in  the  LNG  example)  was  changed.  The  order 
of  the  problems  (LNG  vs.  apartments)  and  the  type  of  range  change 
(increase  vs.  decrease)  was  varied  according  to  Table  4 . Subjects 
were  randomly  assigned  to  one  of  the  four  conditions  in  this  table. 

When  repeating  the  fMART  procedure  subjects  did  not  need  to 
reassess  utility  functions  for  the  unchanged  attributes. 

Therefore  subjects  were  presented  with  the  original  alternative  by 
attribute  matrices  in  Tables  2 and  3 with  only  the  first  attribute 
levels  changed.  They  also  were  presented  with  an  alternative  by 
attribute  matrix  of  their  original  utility  measures  with  only  the 
first  attribute  measures  missing.  Subjects  reassigned  utilities  for 
that  first  attribute  in  both  the  apartment  and  the  LNG  siting  prob- 
lems. Subjects  then  reassigned  all  weights  according  to  the  SMART 
procedure . 

Results 

Of  the  24  subjects,  four  could  not  follow  instructions  and  their 
data  were  disregarded.  The  remaining  20  subjects  provided  as  basic 


TABLE  4 

Experimental  Conditions 
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Condition 

First  Problem 

Range  Change 

Ntnber  of  Subjects 

1 

LNG  Plant 

Increase 

6 

2 

LNG  Plant 

Decrease 

6 

3 

Apartment 

Increase 

6 

4 

Apartment 

Decrease 

6 

15 


data  original  and  revised  weights  and  single  attribute  utility 
functions  in  the  two  evaluation  problems.  To  confute  the  expected 
weight  change  according  to  equation  (5)  three  overlapping  points  of 
the  original  and  the  revised  utility  fuictiore  in  the  first  attribute 
were  used  together  with  the  comerpoint  which  both  utility  functions 
shared.  In  the  range  decrease  condition  T was  computed  as  the  aver- 
age 


T - 


100-u^y^) 

lOO-iyfrj.) 


In  the  range  increase  condition  T was  computed  as 


(8) 


3 loo-u1'(yli) 

1 j-l  lOO-Ujtyj.)  (9) 

where  y^  denotes  the  three  points  that  the  two  utility  functions 
share,  of  equation  (4)  is  in  all  cases  the  coninon  point  of  both 
utility  functions  at  which  they  attain  the  maximum  value  of  100,  i.e., 

UjfXj)  - Uj’fXj)  ■ 100.  (10) 

Table  5 presents  the  original  normalized  weights,  the  revised 
re-normalized  weights  and  the  expected  re-normalized  weights  for 
the  apartment  and  LNG  problems  and  for  the  range  increase  and  the 
range  decrease  separately.  The  normatively  revised  weights  typically 
increase  or  decrease  between  .01  and  .05  (or  about  5-15%)  as  compared 
to  the  original  weights.  Actual  weight  changes  aTe  smaller  and  show  no 
pattern  of  increases  or  decreases  as  predicted  from  the  change  in  range. 
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TABLE  5 

Original  weight  fy),  revised  weight  {Wj#),  and  expected  weight  (Wj'J 
as  a result  of  a change  in  the  range  of  the  first  attribute 
(all  weights  are  re-normalized  to  add  to  1) 


RANGE  INCREASE 

LNG  PROBLEM  (Conditions  1 and  3) 


Subject 

Original 
weight  Wj 

Revised 
weight  Wj* 

Expected 
weight  Wj' 

Direction 
of  change 

1 

.278 

.267 

.328 

wrong 

9 

.278 

.250 

.329 

wrong 

13 

.293 

.281 

.344 

wrong 

17 

.286 

.263 

.354 

wrong 

21 

.270 

.227 

.426 

wrong 

3 

.056 

.056 

.088 

none 

7 

.189 

.206 

.229 

right 

11 

.200 

.273 

.234 

right 

15 

.264 

.273 

.369 

right 

19 

.070 

.133 

* * 

.059 

right 

** 

In  the  number  indicated  by  the  shape  of  the  reassessed  utility 
function  strongly  deviated  from  the  original  utility  fwction, 
and  expected  weignts  based  on  formulas  (8)  and  (9)  were  in  the 
wrong  direction. 


TABLE  5 (continued) 
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RANGE  INCREASE 

APARTMENT  PROBLB4  (Conditions  1 and  3) 


Subject 

Original 
weight  Wj 

Revised 
weight  Wj* 

Expected 
weight  Wj' 

Direction 
of  change 

1 

.189 

.205 

.293 

right 

9 

.217 

.242 

.268 

right 

13 

.310 

.295 

.390 

wrong 

17 

.260 

.257 

.426 

wrong 

21 

.229 

.235 

.402 

right 

3 

.205 

.159 

.232 

wrong 

7 

.028 

.028 

.028 

none 

11 

.185 

.196 

.236 

right 

15 

.221 

.243 

.361 

right 

19 

.214 

.222 

.303 

right 

18 


TABLE  5 (continued) 

RANGE  DECREASE 

LNG  PROBLBl  (Conditions  2 and  4) 


Subject 

Original 
weight  Wj 

Revised 
weight  Wj* 

Expected 
weight  Wj' 

Direction 
of  change 

2 

.277 

.284 

.234 

wrong 

6 

.212 

.257 

.157 

wrong 

10 

.258 

.266 

.240 

wrong 

14 

.225 

.220 

.144 

right 

18 

.337 

.440 

.289 

wrong 

22 

.221 

.227 

.185 

wrong 

8 

.329 

.281 

.246 

right 

12 

.313 

.313 

.309 

none 

20 

.234 

.224 

.197 

right 

24 

.250 

.239 

.207 

right 

19 


TABLE  5 (continued) 


RANGE  DECREASE 


jj 

1 

(Conditions 

2 and  4) 

Subject 

Original 
weight  Wj 

Revised 
weight  Wj* 

Expected 
weight  Wj' 

Direction 
of  change 

2 

.258 

.262 

.230 

wrong 

6 

.313 

.313 

** 

.340 

none 

10 

.219 

.226 

.158 

wrong 

14 

.297 

.233 

.282 

right 

18 

.329 

.392 

.283 

wrong 

22 

.271 

.271 

.207 

none 

8 

.264 

.250 

.193 

right 

12 

.353 

.313 

.352 

right 

20 

.288 

.262 

.173 

right 

24 

.258 

.254 

.225 

right 

** 

In  the  nunber  indicated  by  the  shape  of  the  reassessed  utility 
function  strongly  deviated  from  the  original  utility  function, 
and  expected  weights  based  on  formulas  (8)  and  (9)  were  in  the 
wrong  direction. 
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Apparently  subjects  were  not  even  ordinally  correct 
in  their  reassessment  of  weights.  Table  6 presents  a si^>le  s unary 
of  the  data  in  Table  S to  analyze  whether  subjects  changed  their 
weights  in  the  right  direction  after  a decrease  or  an  increase  in 
the  range  of  attribute  levels.  For  the  LNG  problem  subjects  were 
more  often  wrong  than  right  (in  12  cases  out  of  20).  In  the  apartment 
problem  subjects  fared  a little  better  but  still  9 responses  out  of 
20  were  in  the  wrong  direction.  Overall  subjects  were  not  sensitive 
to  the  range  changes. 

A more  detailed  analysis  shows  that  of  those  subjects  which 
changed  their  weights  in  the  correct  direction,  most  subjects  (83% ) 
did  not  sufficiently  adjust.  Only  two  responses  were  in  the  right  direc- 
tion but  were  of  too  great  a magnitude.  Only  a single  response  out 
of  40  was  correct! 

How  does  this  apparent  range  insensitivity  and  the  subsequent 
misassessment  of  revised  weights  translate  into  actual  utility  order- 
ings? The  maxima  of  utility  theory  are  very  flat  (see,  for  exanple, 
v.  Kintcrfeldt  and  Edwards,  1973)  which  means  that  modest  errors  in 
changing  nunbers  are  tmlikely  to  affect  orderings.  In  only  3 out  of 
40  cases  did  the  use  of  subjectively  revised  weights  lead  to  a "best" 
option  different  from  the  best  option  using  normatively  revised 
weights. 

Discussion 

The  results  of  the  first  experiment  strongly  suggest  that  sub 
jects  could  not  intuitively  appreciate  the  effect  a change  in  range 
should  have  on  the  iaportance  weight  for  an  attribute  Correct  di- 
rections of  weight  changes  occurred  almost  as  often  as  incorrect 
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TABLE  6 

Surma ry  of  Weight  Change  as  a Fine t ion  of  a Change  in  the  Range 

LNG  PROBLEM 
Weight  should 


Weight 

Did 


increase  decrease  Correct  changes 
increase  4 5 

stay  same  1 1 401 

decrease  5 4 


Weight 

Did 


APARTMENT  PROBLEM 
Weight  should 

increase  decrease  Correct  changes 
increase  6 3 

stay  same  1 2 551 

decrease  3 5 


Weight 

Did 


BOTH  PROBLEMS 
Weight  should 

increase  decrease  Correct  changes 
increase  10  8 

stay  same  2 3 481 

decrease  8 9 
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ones.  Most  of  the  correct  changes  were  too  mil.  However,  the 
noroatively  required  changes  were  not  sufficiently  different  from 
the  (incorrectly)  revised  ones  to  produce  different  decisions. 

There  are  several  reasons  why  subjects'  weight  estimates  *ay 
have  been  range  insensitive.  One  reason  obviously  is  that  the  ex- 
pected weight  change  itself  was  not  strong  enough  (typically  less 
than  201)  to  produce  the  desired  effect.  Revisions  may  there- 
fore reflect  random  reassessment  error  due  to  the  neglect  of  a rela- 
tively small  required  change.  Another  reason  is  that  the  task  may 
have  been  too  corplex  to  produce  the  desired  range  effect.  Both 
argunents  call  for  a strong  manipulation  of  the  task  variables  (at- 
tributes and  ranges).  The  second  experiment  was  designed  to  magnify 
necessary  range  effects  in  a very  sinplified  multiattribute  evalu- 
ation problem.  In  this  experiment  utilities  and  original  weights  were 
prespecified  as  those  of  the  "experimenter,"  leaving  the  subject  with 
the  sole  task  of  revising  weights. 

EXPERIMENT  2 
Method 

Subjects.  69  students  participated  in  this  second  experiment.  Again 
subjects  were  psychology  undergraduates  from  the  University  of  Southern 
California.  Participation  rules  were  the  same  as  in  the  first  exper- 
iment. 

Stimuli.  Stimuli  were  three  cars  described  on  two  attributes,  "gas 
mileage"  and  "weight."  Subiects  saw  them  in  the  form  of  a car  by  at- 
tribute matrix  as  in  Table  7.  The  range  of  the  mileage  attribute  was 
changed  by  multiplying  all  nusbers  by  2 (the  changed  attribute  levels 
are  given  in  brackets). 
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TABLE  -7 

Stimuli  for  the  Car  Evaluation  Problem  in  Experiment  2 
(In  Brackets:  Oianged  Values  for  Range  Change) 


CARS 

A 

B 

C 

Gas  mileage 

30 

20 

10 

(riles/gallon) 

(60) 

(40) 

(20) 

Weight 

(poinds) 

2000 

1S00 

2S00 

E 

S 


Procedure.  Subjects  were  introduced  to  the  ideas  of  the  SMART 


technique,  and  were  presented  with  the  "experimenter’s"  single 
dimensional  utilities.  They  were  further  told  that  the  "experiment- 
er" considered  gas  mileage  as  twice  as  important  as  weight,  given 
the  range  of  mileage  available,  thus  giving  a normalized  weight  of 
.667  to  mileage  and  one  of  .333  to  weight.  34  subjects  then  re- 
ceived the  following  instructions: 

"Now  suppose  that  someone  discovered  a way  to  double  gas  mileage 
so  that  the  values  for  cars  A,  B,  and  C,  are  now  60,  40,  and  20. 
For  me,  the  rescaled  values  (location  measures)  will  be  as  they 
were  before,  100,  50,  0 respectively.  What  should  my  importance 
nimbers  (relative  weights)  now  be?  Should  they  remain  at  .667 
and  .333  or  should  they  change?  If  they  should  change,  what 

should  they  change  to?  Change?  (yes  or  no) 

If  yes,  to  what?  (mileage) and  (weigh:) 

respectively." 

35  subjects  received  slightly  different  instructions  which  re- 
quired a change  in  weights  and  stressed  ratio  assessment: 

"Now  suppose  that  someone  discovered  a way  to  double  gas  mile- 
age so  that  the  values  for  cars  A,  B,  and  C are  now  60,  40,  and 
20.  For  me,  the  rescaled  values  will  be  as  they  were  before, 

100,  50,  and  0 respectively.  But  now  these  numbers  refer  to  dif- 
ferent actual  gas  mileages  than  they  did  before.  Presumably , this 
has  not  changed  my  feeling  about  the  importance  of  any  specific 
gas  mileage  as  compared  with  a specific  weight.  If  so,  then  I 
should  change  the  nvmbers  that  represent  the  importance  of  gas 
mileage  to  allow  for  the  changed  mileage  aspects  of  the  new  set 
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of  cars.  Should  I increase  or  decrease  the  iiportance  of  gas 

mileage?  Instead  of  2/1  ratio  of  mileage 

importance  to  weight  importance,  what  ratio  should  I use? 

What  should  the  importance  numbers  be  if  they  should  sun  to  1? 

5 


Results 

Table  8 summarizes  the  results  of  experiment  2.  In  the  first 
instruction  group  24  subjects  indicated  that  there  should  be  no 
change  in  weights,  clearly  an  error.  Of  the  remaining  10  subjects, 

8 answered  in  the  correct  direction.  One  answered  in  the  wrong 
direction.  One  did  not  indicate  what  the  change  should  be.  Of 
the  eight  that  responded  with  the  correct  direction,  five  gave  ex- 
actly the  correct  answer;  one  erred  in  not  renormalizing  the 
weights;  and  the  remaining  two  subjects  gave  slightly  underestimated 
responses . 

In  the  second  instruction  group  thirteen  of  the  35  subjects  were 
livable  to  give  importance  weights  which  summed  to  one.  No  analysis 
of  their  responses  is  appropriate.  Although  the  response  format  did 
not  permit  this,  two  subjects  indicated  that  there  should  be  no 
change  in  weights.  Of  the  remaining  21  subjects  two-thirds  believed 
that  the  change  should  be  in  the  wrong  direction,  and  one- third  gave 
responses  in  the  correct  direction.  But  only  one  of  the  subjects 
gave  the  correct  response,  while  the  rest  were  conservative. 

Discussion 

Even  in  an  absurdly  simple  problem  subjects  apparently  had  prob- 
lems appreciating  the  sensitivity  of  importance  weights  to  a change 
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TABLE  I 

Suasary  of  Experiment  2:  Changes  in  Weight  as  a Function 
of  a Range  Increase  and  Instructions 


Instruction  grotp  1 
(normalized  weights) 


Instruction  grot?  2 
(ratio  weights) 
Forced  Choice 


Exactly 

Correct 

Response 

5 

1 

Correct 

Direction 

But  Not 

Sufficient 

3 

6 

No  Change 

24 

2 

Wrong  Direction 

1 

14 

Not  Analyzed 

1 

13 

Total 

34 

36 
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in  the  range  of  an  attribute.  In  the  first  set  of  instructions  701 
of  the  subjects  felt  that  no  change  in  the  weights  should  take  place. 
This  supports  the  results  of  the  first  experiment  which  used  more 
conplex  stimuli.  The  insufficient  adjustment  effect  of  the  first 
experiment  did  not  occur  inder  these  instructions,  probably  because 
the  problem  was  so  sinple. 

The  second  set  of  instructions  seems  to  have  confused  subjects. 
Perhaps  the  words  "presumably  this  (change  in  the  gas  mileage 
levels)  has  not  changed  my  feelings  about  the  importance  of  any 
specific  gas  mileage  as  compared  to  a specific  weight”  were  con- 
fusing. In  general,  the  results  of  this  instruction  group  also 
sipport  the  hypothesis  that  subjects  are  range  insensitive  when 
revising  importance  weights,  with  only  7 subjects  responding  in 
the  right  direction. 

Both  experiments  show  that  subjects  did  not  revise  weights  after 
a change  in  the  range  of  an  attribute  as  would  be  predicted  by  the 
normative  rule.  The  data  suggest  that  subjects  instead  were-  rather 
insensitive  to  the  change  in  range. 

The  meaning  of  inportance  as  a relatively  range  insensi- 
tive attribute  characteristic  may  have  distracted  subjects  from  pro- 
perly considering  ranges  in  their  weight  assessment.  "Inportance" 
is  a relatively  uiexplored  psychological  concept.  Several  uses  of  the 
word  indicate,  however,  that  it  is  a rather  stable  property  of  attri- 
butes, which  is  carried  over  from  one  situation  (and  one  range)  to 
another.  If  no  alternatives  are  specified  people  can  usually  give  a 
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ranking  of  attributes  in  order  of  i^wrtance.  In  fact,  such  a range 
independent  assessment  was  originally  suggested  in  Edwards  (1972). 

The  fact  that  people  can  give  inport  nice  orderings  without  specified 
alternatives  and  ranges  may  mean  that  they  have  sor  ? plausible  set 
of  alternatives  and  ranges  in  mind,  when  judging  importance . Accord- 
ing to  this  interpretation  the  importance  judgments  should  only 
change  when  the  environment  radically  changes  the  plausible  set  of 
alternatives. 

If  importance  is  in  fact  relatively  stable  across  situations,  the 
experimental  instructions  may  have  induced  two  opposing  reactions  in 
the  subjects:  on  one  hand  their  intuitive  appreciation  of  the  concept 
of  "importance"  would  suggest  no  change,  irtiile  the  explicit  range 
effect  instructions  required  change.  These  opposing  trends  may  have 
confused  subjects,  as  the  high  rate  of  obviously  in  congruent  answers 
suggests.  An  obvious  manipulation  to  test  this  hypothesis  is  to  leave 
out  the  label  "importance"  altogether,  e.g.,  by  asking  "how  much  more 
would  you  like  to  step  from  the  worst  level  in  attribute  1 to  the 
best  as  coepared  to  stepping  from  the  worst  in  attribute  2 to  the 
best?" 

The  results  of  these  experiments  should  be  inter- 
preted with  much  caution.  If  this  experiment  had  shown  that  subjects 
are  range  sensitive  in  their  weight  assessment,  a major  problem  in 
applying  the  SMAET  technique  would  have  been  solved.  One  problem  for 
SMART  sieges  ted  by  the  experimental  results  is  that  instructions  which 
cotple  iiportance  judgments  with  ranges  in  attribute  levels  can  be 
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confusing.  SM^RT  procedures  with  swing  weight  techniques  should  con- 
sequently be  done  very  carefully.  The  hypothesis  that  "import  - 
ance"  is  a range  insensitive  concept  poses  a major  problem  to  the 
SWVRT  procedure.  But  it  also  offers  two  alternative 

solutions:  In  the  first,  the  term  importance  would  be  given  up  al- 
together and  substituted  by  cross  attribute  relative  value  or  in- 
difference judgments  as  in  Keeney  and  Raiffa  (1976).  In  the  second, 
importance  judgments  would  be  made  independently  of  ranges  and 
ranges  would  be  defined  to  cover  a "plausible"  set  of  alternatives 
rather  than  the  available  set. 

While  the  first  solution  is  simple,  and  is  easily  implemented 
with  only  minor  rewording  of  the  weighting  procedure,  the  second  so- 
lution requires  a quite  different  view  of  the  nultiattribute  utility 
modeling  problem.  If,  in  fact,  "importance"  is  a judgment  which 
has  substantive  meaning  and  can  numerically  be  scaled  independently 
of  its  interpretation  as  a rescaling  factor,  incorporating  importance 
could  give  additional  substance  to  a multiattribute  evaluation  model. 
This  argument  is  not  unlike  the  argument  for  using  external  judgments 
of  probabilities  as  independent  inputs  into  an  expected  utility  model 
(as  opposed  to  the  interpretation  of  probabilities  as  prices  or  rates) . 
The  problem,  of  course,  remains,  how  to  standardize  single  attribute 
utility  functions  in  such  substantive  uses  of  importance  judgments. 
This  is  clearly  an  experimental  question. 
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cannot  be  used  to  match  standardized  single  attribute  utility  functions.  To 
examine  the  range  sensitivity  of  i^xnrtance  weight  judgaents  two  experiments 
were  designed.  In  the  first  experiment  college  students  gave  relative 
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'importance  weight  judgments  for  a nurtwr  of  attributes  when  evaluating 
apartments  and  liquified  natural  gas  plant  locations.  After  the  initial 
importance  weight  assessments  the  range  of  alternatives  in  one  attribute 
was  changed  and  subjects  reassessed  their  weights.  Although  subjects 
were  explicitly  instructed  to  take  ranges  into  accouit  when  making  these 
judgments,  they  were  inable  to  adjust  their  weights  appropriately.  To 
magnify  possible  range  effects  a second  experiment  examined  a very  sinple 
two  attribute  car  evaluation  problem.  Subjects  were  asked  directly  if 
weights  should  change  after  the  range  in  one  attribute  was  doubled.  Most 
subjects  indicated  that  there  should  be  no  change.  The  results  of  these 
experiments  suggest  that  subjects  have  plausible  ranges  in  mind  when 
assessing  inportance  weights  and  that  they  are  mailing  to  change  weights 
after  relatively  spurious  changes  in  the  alternative  set. 
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